An experimental study of the uses and misuses of null hypothesis significance tests among psychologists and statisticians

In one of the first experiments about the use of NHST (Rosenthal and Gaito, 1963), researchers in psychology were asked expressions of degree of belief in the hypothesis of an effect as a function of the associated p levels and sample sizes. The authors emphasized a “cliff effect” for the .05 level for which the degree of belief was quite the same as for the .03 level, and came to the conclusion that it was an effect of the norm, due to the “sacred” reference to the .05 level. 18 psychological researchers were submitted to a replication of this experiment (Poitevineau, 1998). Results The mean average findings confirm those previously obtained. But, as a matter of fact, the study of individual curves revealed that the attitude of researchers toward p values was not so homogeneous as it could be conceived. Indeed subjects can actually be classified into three categories: (1) 10 subjects with a decreasing exponential curve; (2) 4 subjects with a negative linear curve; (3) 4 subjects with an all-or-none curve having a very high degree of belief when p<.05 and nearly a null degree of belief otherwise. Clearly, this last minority category was largely responsible for the “cliff effect” of the average curve. Thus, not only researchers' attitudes toward p values were heterogeneous, but most researchers expressed graduated judgments, contrasting with the common practice to treat a test outcome as a dichotomy (significant vs nonsignificant) in publications.